Add GB200 DSv4.1 Flash Mooncake offload / 新增 GB200 Mooncake 卸载 - #2996
Oseltamivir wants to merge 5 commits into
Conversation
新增 GB200 DSv4.1 Flash Mooncake 嵌入式 DRAM KV 卸载配方，包含既有并发及更高并发；保留完整 AgentX 轨迹、Engram UVA 和黄金合成 AL 3.51。为 Engram 和传输缓冲区预留主机内存，并在退出时清理服务进程。
补充 Mooncake 卸载配方的性能变更 PR 链接。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| @pytest.mark.parametrize("occupied", [False, True]) | ||
| def test_select_available_server_port_avoids_an_existing_listener(occupied: bool) -> None: | ||
| with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener: | ||
| listener.bind(("0.0.0.0", 0)) |
| else: | ||
| assert selected == preferred | ||
| with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server: | ||
| server.bind(("0.0.0.0", selected)) |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| "protocol": "rdma", | ||
| "device_name": os.environ.get("MOONCAKE_DEVICE_NAME", ""), | ||
| "enable_offload": False, | ||
| }, output, indent=2) |
There was a problem hiding this comment.
Mooncake RDMA devices left unspecified
Medium Severity
The embedded Mooncake config sets protocol to rdma with an empty device_name, and the GB200 srun does not pass MOONCAKE_DEVICE_NAME or mount InfiniBand devices. The B200 Mooncake recipe pins specific mlx5_* NICs. Auto-picking every RDMA device on a GB200 tray can fail handshake or GPUDirect, so KV offload never becomes ready.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 845418b. Configure here.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - dsv41flash-fp4-gb200-vllm-agentic-dspark-mooncake | ||
| description: | ||
| - "Add GB200 DSv4.1 Flash embedded Mooncake DRAM KV offload, golden synthetic AL 3.51 and concurrency 1-512" | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 New changelog entry uses pr-link: TBD, which fails validate_perf_changelog.py's gate: on a PR run the link must exactly match the PR number URL or be an XXX placeholder, and on main it must match the canonical .../pull/<digits> regex — "TBD" matches neither. Fix: replace with the actual PR URL once known, or the XXX/https://.../pull/XXX placeholder accepted by validate_added_pr_link.
Extended reasoning...
utils/validate_perf_changelog.py:147-163 (validate_added_pr_link) checks link not in PR_LINK_PLACEHOLDERS and link != expected for PR runs, and CANONICAL_PR_LINK.fullmatch(link) for main-branch runs; PR_LINK_PLACEHOLDERS only contains "XXX" and the XXX URL form, not "TBD". Since this PR adds pr-link: TBD for the new dsv41flash-fp4-gb200-vllm-agentic-dspark-mooncake entry, the changelog validation CI gate raises ChangelogValidationError ("new PR entry must use ... or an XXX placeholder") and blocks merge until the placeholder is corrected.
Verification: normal. The PR appends a new changelog entry (perf-changelog.yaml lines 7166-7170) whose pr-link is literally TBD. This entry is an addition (it comes after all base entries), so compare_entries reaches validate_added_pr_link(str(entry.get("pr-link") or ""), pr_number) at validate_perf_changelog.py:209. In validate_added_pr_link (147-163): on a main-branch run (pr_number is None),…
| done | ||
|
|
||
| # Pyxis shares the host network; port 8888 can already belong to a host service. | ||
| select_available_server_port |
There was a problem hiding this comment.
🟡 (optional) select_available_server_port() is called long before the actual vllm serve bind: setup_agentic_mooncake() (pip install + up to 30s master-readiness polling) runs afterward, reopening the port race the helper's own comment says to avoid ('Call immediately before server launch'). On the shared host network namespace with concurrent jobs, another process can grab the freed probe port during that window, so vllm serve fails to bind late, after model download/install. Fix: move the port probe to immediately precede the vllm serve launch (after setup_agentic_mooncake), or re-verify/re-bind right before use.
Extended reasoning...
benchmark_lib.sh's select_available_server_port() binds to a port, immediately closes the socket, and exports PORT — it never holds the port, so the doc comment mandates calling it 'immediately before server launch' to minimize the collision window. In this script it's called at line 41, but setup_agentic_mooncake() (line 63) then runs agentic_pip_install (network/pip time) and polls mooncake_master readiness up to 30 times at 1s each before returning, and only after that does vllm serve actually bind the port at line 89. Since Pyxis shares the host network namespace across concurrent matrix jobs (per this script's own comment), any other job on the host probing/binding ports during that multi-second-to-tens-of-seconds gap can take the freed port, causing 'vllm serve --port "$PORT"' to fail to bind — the job fails late, after the expensive model download and Mooncake install, instead of the race being minimized as the helper's contract intends.
Verification: nit. The mechanism is real and demonstrable from the code. select_available_server_port() (benchmark_lib.sh:28-45) binds inside with socket... as sock:, prints the port number, then the context manager closes the socket and releases the port — only PORT (a number) is exported, the port is not held. Its doc comment (benchmark_lib.sh:27) states: "Call immediately before server launch and… | nit.…
| NUM_SPEC_TOKENS=5 | ||
| CAPTURE_SIZE=1 | ||
| while (( CAPTURE_SIZE < CONC * (1 + NUM_SPEC_TOKENS) && CAPTURE_SIZE < 2048 )); do | ||
| CAPTURE_SIZE=$((CAPTURE_SIZE * 2)) |
There was a problem hiding this comment.
🟡 (optional) CAPTURE_SIZE is capped at 2048 while the graph must cover CONC*(1+NUM_SPEC_TOKENS)=CONC6 tokens; for the highest declared concurrency (512) that's 3072, so --max-cudagraph-capture-size silently ends up smaller than the actual max batch, unlike lower concurrencies where the cap never binds. Fix: raise the cap (or warn/fail) whenever CONC(1+NUM_SPEC_TOKENS) exceeds the hard ceiling, so conc=512 doesn't silently fall back to eager execution for its largest batches with no log message.
Extended reasoning...
NUM_SPEC_TOKENS=5, so the loop target is CONC*6. For CONC in the config's conc-list up to 256, target<=1536 and the doubling loop overshoots to 2048 harmlessly. For CONC=512 (explicitly in the new recipe's conc-list [1,2,4,8,16,32,64,128,256,512]), target=3072 but the && CAPTURE_SIZE < 2048 clause stops doubling at 2048, so --max-cudagraph-capture-size 2048 is passed to vLLM even though decode batches at conc=512 with 5 speculative tokens can reach 3072 tokens. vLLM has no captured graph for batches beyond max_cudagraph_capture_size and runs them eagerly, degrading throughput specifically at the top of the newly-added concurrency sweep with no warning printed, undermining the script's own comment 'size graph capture for the sweep'.
Verification: nit. The arithmetic and reachability check out. Line 34-38: NUM_SPEC_TOKENS=5, CAPTURE_SIZE=1, while (( CAPTURE_SIZE < CONC * (1 + NUM_SPEC_TOKENS) && CAPTURE_SIZE < 2048 )). For CONC=512 the target is 512*6=3072; the doubling sequence reaches 2048 and the CAPTURE_SIZE < 2048 clause halts it there, so --max-cudagraph-capture-size 2048 (line 84) is passed even though the decode… | nit.…
修正 Mooncake 主机内存单位，限制卸载模式的图捕获内存，并保留启动失败诊断。
同步 main 并保留追加的性能变更记录。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34557398402 |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
There are 3 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b16fd26. Configure here.
| --max-model-len 1048576 | ||
| --max-cudagraph-capture-size "$CAPTURE_SIZE" | ||
| --disable-uvicorn-access-log | ||
| ) |
There was a problem hiding this comment.
High concurrency left uncapped
Medium Severity
The vLLM command never sets --max-num-seqs, while the recipe advertises concurrency 256 and 512. vLLM’s default scheduler cap is much lower, so those sweep points cannot actually run the configured number of live sessions. CUDA graph capture is scaled for CONC, but the scheduler budget is not.
Reviewed by Cursor Bugbot for commit b16fd26. Configure here.
| --container-mounts="$GITHUB_WORKSPACE:/ix,$HF_HUB_CACHE_HOST_PATH:/hf-cache" \ | ||
| --no-container-mount-home --container-remap-root \ | ||
| --container-workdir=/ix --no-container-entrypoint \ | ||
| --export=ALL,PORT=8888 bash "$BENCH_SCRIPT" |
There was a problem hiding this comment.
AgentX cache mount omitted
Medium Severity
The new single-tray srun only bind-mounts the workspace and HF hub cache. The same GB200 launcher already mounts the shared AIPerf mmap cache for other AgentX jobs because the full semianalysis_cc_traces_weka_062126 corpus is tens of gigabytes and is re-tokenized from scratch without it. Each concurrency point will redo that work.
Reviewed by Cursor Bugbot for commit b16fd26. Configure here.
将 GB200 主机内存预算从 60% 降为 40%，缓解注册时的 OOM，并记录 NUMA 内存诊断。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34561513231 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34565285925 |
2 participants
Add GB200 TP4 DSv4.1 Flash AgentX with embedded
MooncakeStoreConnectorDRAM KV offload, Engram UVA and golden synthetic AL 3.51. Concurrencies: 1, 2, 4, 8, 16, 32, 64, 128, 256, 512. Uses fullsemianalysis_cc_traces_weka_062126traces. Host allocation uses 40% DRAM (360 GB), reserving Engram and transfer buffers; eval keeps real verification. Follows the upstream Mooncake recipe. Local matrix, launcher, serving-command and Mooncake setup tests pass; GPU validation pending.新增 GB200 TP4 DSv4.1 Flash AgentX，启用嵌入式
MooncakeStoreConnectorDRAM KV 卸载、Engram UVA 和黄金合成 AL 3.51。并发：1, 2, 4, 8, 16, 32, 64, 128, 256, 512。使用完整semianalysis_cc_traces_weka_062126轨迹；主机内存预算为 DRAM 的 40%（360 GB），预留 Engram 和传输缓冲区，eval 保留真实验证。遵循上述上游 Mooncake 配方。本地矩阵、launcher、服务命令和 Mooncake 启动测试通过，GPU 验证待完成。Note
Medium Risk
New GPU benchmark path with complex host-memory budgeting and KV offload orchestration; GPU validation is still pending and mis-sized budgets could cause OOM or invalid sweep results.
Overview
Adds DeepSeek-V4.1-Flash (
dsv41flash) to the model matrix and a new GB200 AgentX sweep keydsv41flash-fp4-gb200-vllm-agentic-dspark-mooncake: TP4, fp4 vLLM with DSpark MTP, Engram CPU offload, and optional embedded Mooncake DRAM KV offload at concurrency 1–512 and 1M context.Shared benchmark plumbing gains
select_available_server_port(host-network safe) andsetup_agentic_mooncake(byte-accurate host budget after Engram reserve, CUDA13 wheel selection, embedded master +MooncakeStoreConnector). The newdsv41flash_fp4_vllm_mtp.shrecipe wires these in, caps CUDA graph capture at 512 when Mooncake is on, and logs NUMA/OOM diagnostics on failure.launch_gb200-nv.shadds a direct single-tray Pyxis path for this model. Docs/changelog document the conservative 40% host DRAM budget after OOM at higher allocation. Unit tests cover Mooncake setup, graph limits, port selection, and GB200 launchersrunwiring.Reviewed by Cursor Bugbot for commit 4f6483f. Bugbot is set up for automated code reviews on this repo. Configure here.